Pre-emptively initializing an automated assistant routine and/or dismissing a scheduled alarm

ABSTRACT

Implementations set forth herein relate to initializing performance of an automated assistant routine and/or dismissing an alarm pre-emptively according to satisfaction of one or more conditions. A condition can be satisfied by a user acknowledging the alarm when the alarm is going off, or causing the alarm to be dismissed prior to a time at which the alarm was scheduled for. The user can cause the alarm to be dismissed pre-emptively by interacting with the automated assistant prior to the time the alarm was scheduled for and/or interacting with a device, which is known to the automated assistant, prior to the time that the alarm was scheduled for. In this way, actions that cause an alarm to be dismissed can be recognized and used to initialize other processes, such as an automated assistant routine, thereby reducing a number of inputs needed from a user.

BACKGROUND

In certain contexts, a user may request an alarm be triggered at acertain time in order to wake the user, so that the user can wake up tostart their morning routine. Should the user wake before their alarmgoes off, their alarm may nonetheless still be triggered at thescheduled time. Depending on the type of alarm (e.g., audio and/orvisual), power and computational resources may be wasted on generatingthe alarm—which was not necessary since the user was already awake.Furthermore, the user may have to interrupt their routine in order tostop the alarm, thereby interrupting whatever tasks they may have beeninvolved with before stopping the alarm. Similar mistakes can occur whenthe user is interacting with an automated assistant, which may bescheduled to initialize particular actions at certain times.

In many cases, before the automated assistant can interpret and respondto a user's request, it must first be “invoked,” for example, usingpredefined oral invocation phrases that are often referred to as “hotwords” or “trigger phrase.” Thus, many automated assistants operate in a“default listening state” in which they are always “listening” to audiodata sampled by a microphone for a limited (or finite, or “default”) setof hot words. Any utterances captured in the audio data other than thedefault set of hot words are ignored. Once the automated assistant isinvoked with one or more of the default set of hot words, it may operatein what will be referred to herein as a “speech recognition state”wherein, for at least some time interval after invocation, the automatedassistant performs speech-to-text (“STT”) processing of audio datasampled by a microphone to generate textual input, which in turn issemantically processed to determine a user's intent (and to fulfill thatintent).

In certain contexts, a user may not be able to comfortably provide atrigger phrase, and therefore may elect not to invoke an automatedassistant, such as when the user is the first one to wake in a householdwhere others are sleeping. Furthermore, it may be computationallyinefficient to always have an automated assistant operating in the“listening” state. For instance, the “listening” state may require oneor more computing devices to be constantly sampling audio data that isbeing generated based on a microphone output. Compared to a simple pushbutton input, such constant monitoring of a microphone output can becomputationally intensive, and may also consume network bandwidth—shouldthe audio data be processed at a separate server device.

SUMMARY

The present disclosure is generally directed to methods, apparatus, andcomputer-readable media (transitory and non-transitory) for dismissing ascheduled operation, to be performed by a computing device, in responseto one or more conditions being satisfied. Furthermore, an automatedassistant can be invoked for performing one or more actions (e.g., aroutine that includes multiple actions) in response to the scheduledoperation being dismissed. In this way, a number of user inputs, whichwould otherwise be processed by one or more computing devices, can bereduced. For instance, by triggering an automated assistant routinepre-emptively in response to a user dismissing an alarm (e.g., bytapping a button of a device), the user would not necessarily have toprovide a subsequent spoken utterance for triggering the automatedassistant routine. Omitting the subsequent spoken utterance in this waycan preserve network bandwidth, as voice data would not need to betransmitted to a separate server for processing. Furthermore, bypassingprocessing data corresponding to the spoken utterance can preserveprocessing bandwidth, as a processor of the device receiving the spokenutterance would not have to cache and/or compress any data correspondingto the spoken utterance.

In some implementations, a user can have multiple devices connectedwithin an environment such as their home. Each device of the multipledevices can provide access to an automated assistant, which can beinvoked in response to the user providing a natural language input, suchas a spoken utterance or trigger phrase. The automated assistant canperform actions or sets of action (e.g., routines) that can includecreating or modifying certain settings, such as establishing an alarmand causing the alarm to go off at a particular time. For instance, theuser can provide a spoken utterance to an automated assistant interfaceof a computing device in their bedroom by speaking, “Assistant, pleaseset my alarm for 6:00 AM tomorrow morning.” In response, the automatedassistant can be invoked from the computing device and subsequently setthe alarm. When the alarm is going off at 6:00 AM the next morning, anoutput (e.g., music) can be dismissed by the user via an explicit inputfor dismissing the output, such as a button tap or another spokenutterance. However, in order to prevent the alarm from going off whenthe user is already awake prior to the alarm, the alarm can bepre-emptively dismissed.

In order to pre-emptively dismiss the alarm, a determination of whethera situational condition has been satisfied can be made, and, when thesituational condition is satisfied, the alarm can be pre-emptivelydismissed. For instance, a situational condition can include a parameterthat requires an indication that the user is awake, active, or otherwisenot sleeping at least within a finite period of time prior to ascheduled time for the alarm to go off. As an example, the situationalcondition can be satisfied when the user has activated a light in theirhome prior to a time when the user has scheduled the alarm to go off,but after a time that the user has been sleeping for a threshold periodof time. In this way, should the user get up prior to their alarm goingoff, and switch on a light, an output scheduled to be emitted at thescheduled time for the alarm can be omitted or bypassed.

Alternatively, or additionally, when the alarm is going off (e.g.,rendering audio and/or video capable of being perceived by the user),the alarm can be dismissed when the situational condition is satisfied.As an example, the situational condition can be satisfied when the userhas activated a podcast application and/or speaker in their home after atime that the user has been sleeping for a threshold period of time.Therefore, should the user get up when their alarm is going off, andactivate the podcast application and/or speaker in their home, an outputcorresponding to the alarm going off can be dismissed. In someimplementations, the alarm can continue to go off (e.g., render audioand/or video capable of being perceived by the user) when thesituational condition is not satisfied. For example, when a temporalarrival of the scheduled time for the alarm occurs, and the alarmsubsequently goes off, the user may not perform actions that cause thesituational condition to be satisfied. As a result, the alarm cancontinue to go off until the situational condition, or anotherapplicable condition, is satisfied.

In some implementations, a situational condition can be based on one ormore interactions, between the user and the automated assistant, thatpreviously occurred within a period of time of a past occurrence of anevent, or in a manner in which a relationship between the one or moreinteractions and the event could be determined. For instance, asituational condition can be generated for any of the implementationsdiscussed herein when a user exhibits a habit of performing one or moretasks around the time that a particular alarm has been scheduled to gooff. As an example, the automated assistant can detect that the user,upon waking up, typically moves about their house, turns on theirkitchen lights, and starts listening to music. This series of tasks setforth by the user can be used as a basis for establishing a situationalcondition, for example, in order to pre-emptively dismiss an alarm. Inthis way, should the user perform the series of tasks prior to theiralarm going off, or when their alarm is going off, the situationalcondition can be considered satisfied, and the alarm can be dismissed(or pre-emptively dismissed). In some implementations, generatingparameters for defining the situational condition can be based on anumber of times one or more users have performed one or more tasks inthe past, a context in which the user has performed the one or moretasks, one or more events related to the one or more tasks, and/or anyother information suitable for use when identifying tasks relevant to ascheduled event. For instance, although a user may explicitly requesttheir alarm be dismissed when they invoke the automated assistant todismiss the alarm, the automated assistant can also generate asituational condition for dismissing the alarm based on the user, atleast a threshold number of times (e.g., at least 10 times) in the past,having turned on their lights and adjusted their thermostat after theiralarm had gone off.

In some implementations, the user can also request that the automatedassistant perform a routine when their alarm goes off or is dismissed.The routine can include one or more actions to be performed by theautomated assistant, and can be previously configured by the user duringa dialog session between the user and the automated assistant. Forexample, the user can request that the automated assistant perform a“morning” routine by providing a spoken utterance such as, “Assistant,when I say ‘good morning,’ turn on my coffee maker, play my morningplaylist, and turn on my kitchen lights.” Therefore, when the userrequests that the automated assistant set an alarm, the user can alsorequest that the automated assistant perform the “morning” routine. Thespoken utterance for establishing the aforementioned setting can be, forexample, “Assistant, set my alarm and perform my ‘good morning’ routinewhen my alarm goes off.” As a result, the automated assistant routinecan be set to be initialized when a user input condition has beensatisfied and/or an event condition has been satisfied. For instance, auser input condition can be satisfied by the user providing an inputthat is directed at dismissing the alarm when the alarm is going off.The event condition can be satisfied when a temporal arrival of an event(e.g., a time for the scheduled alarm has temporally arrived)corresponding to the event condition has occurred. Therefore, wheneither the alarm has gone off and/or the user has dismissed the alarmwith a particular user input (e.g., a spoken utterance or a gesture toan interface of a computing device), the “morning” routine can beinitialized by the automated assistant.

In some implementations, the user can request that the automatedassistant set the alarm and perform a routine when the alarm goes off oris dismissed, and the automated assistant can perform the routine whenthe alarm is pre-emptively dismissed and/or a situational condition issatisfied. In other words, despite the user requesting the routine beperformed in response to the alarm going off or the user dismissing thealarm, the routine can still be initialized prior to the time scheduledfor the alarm to go off. As an example, the user can set the alarm to gooff at 6:00 AM the next morning, and request that the “good morning”routine be performed when the user dismisses the alarm. However, if theuser awakes prior to 6:00 AM and interacts with the automated assistantby providing a spoken utterance without initializing the “good morning”routine (e.g., “Assistant, what's the weather?”), the automatedassistant can be on notice that the user is awake prior to the time forthe alarm (i.e., 6:00 AM), thereby satisfying the situational condition.As a result of the automated assistant acknowledging that the user isawake prior to the time scheduled for the alarm to go off, the automatedassistant can initialize performance of the routine and cause an output,corresponding to the alarm, to be dismissed or omitted. Therefore, whena temporal arrival of the time for the alarm (e.g., 6:00 AM)subsequently occurs, the output would not be provided by the computingdevice and the automated assistant would have already initializedperformance of the routine.

Pre-emptively affecting a status of particular operations in this way,without necessarily requiring a user input to be processed, can preservecomputational resources that would otherwise be used to processduplicative user inputs. For instance, in scenarios where the user wouldhave to provide a spoken utterance to dismiss their alarm—in order tocause the alarm to be dismissed and an automated assistant routine to beinitialized—the spoken utterance may be processed by one or moredevices, using various speech processing techniques. However, because aninput, which is not directly intended to affect the alarm and/or theautomated assistant routine, can, nonetheless, be used to dismiss thealarm and initialize the automated assistant, the need to store,transmit, and/or process the user input (e.g., a spoken utterance) canbe eliminated.

In some implementations, an ongoing operation can be modified when oneor more conditions, set forth by a user, are satisfied. An ongoingoperation can be an operation that is currently being performed by oneor more computing devices, and optionally via an automated assistant.For instance, a user can request that an automated assistant provideaudio playback of certain media, and when the media is being audiblyrendered for a user, the playback of the media can be considered anongoing operation. Alternatively, or additionally, when the userrequests that an automated assistant set an alarm for a particular time,and the alarm subsequently goes off at that particular time, the alarmbeing audibly rendered can be considered an ongoing operation. Asdiscussed herein, an ongoing operation can be modified when one or moreconditions, which have been specified by a user, are satisfied duringthe ongoing operation. In some implementations, the one or moreconditions can be specified by a user, established by an automatedassistant, pre-configured at one or more computing devices, learned overtime by one or more computing devices using interaction datacorresponding to interaction(s) between one more users and the one morecomputing devices, and/or otherwise can be characterized by dataaccessible to an automated assistant.

In some implementations, a user can request that the automated assistantset an alarm by providing a spoken utterance such as, “Assistant, set analarm for 6 A.M. tomorrow morning, and dismiss the alarm when I turn onthe news in the living room.” In response to receiving the spokenutterance, the automated assistant can cause an alarm to be set for thespecified time, and generate data characterizing the condition fordismissing the alarm. Specifically, the automated assistant can cause asetting to be established for dismissing the alarm when the user turnson the news in their living room using one or more devices in theirliving room. As a result, when the alarm is audibly rendered the nextday at 6 A.M., the audible rendering of the alarm will be dismissed orstopped when news media is being accessed and/or rendered at one or morecomputing devices in a living room of a home of the user. In someimplementations, a status of one or more devices can be accessed inresponse to the alarm going off in order to determine whether one ormore conditions associated with dismissing the alarm have beensatisfied. Alternatively, or additionally, statuses of a subset ofdevices associated with the condition (e.g., devices within the livingroom, rather than the entire home) can be determined specifically whenthe alarm is going off, and/or in response to the alarm going off.Therefore, when a status of one or more devices of the subset of devicescorresponds to a playback of news media, the automated assistant cancause the rendering of the alarm to be dismissed, as requested by theuser via the spoken utterance.

In some implementations, modification of ongoing operations can be atleast partially contingent upon determining which particular user isattempting to satisfy the condition(s). In other words, in someimplementations, in order for an ongoing operation to be modified, onemore computing devices can require that a specified user be the oneperforming certain actions in order to satisfy one or more conditions.As an example, a user that caused an ongoing operation to occur—and alsoset forth the one or more conditions for modifying the ongoingoperation, can be the user that must satisfy the one or more conditionsin order for the ongoing operation to be modified. For instance, a firstuser and a second user can be associated with a particular assistantdevice within a home, and the first user can provide the assistantdevice with a spoken utterance such as, “Assistant, when I return homeand turn on my lights, play my 90's playlist.” In response to receivingthe spoken utterance, the assistant device can generate a setting inwhich the assistant device renders audible media corresponding to a 90'splaylist when the first user returns to their home and turns on thelights. In order to verify whether the first user—and not the seconduser, has returned home—data specifically associated with one or moreusers (e.g., the first user and/or the second user) can be accessed inorder to verify whether the first user has returned home. Such data can,with permission from the user, include GPS data provided by a portableelectronic device, such as a cell phone, security alarm data associatedwith the home of the first user and the second user, vehicle dataassociated with the first user, client data accessible via one or moreclient devices accessible to the first user, and/or any other data withwhich a user can indicate their presence to an automated assistant.

In some implementations, a voice signature of the user, or voiceidentification of the user, can be identified to determine whether aparticular user is home and/or is otherwise acting in furtherance ofsatisfying one or more conditions. For example, when the user arriveshome, the user may provide a spoken utterance to the automatedassistant, and, in response, the automated assistant can determine thata voice characteristic of the user corresponds to a voice signature ofthe first user. Therefore, at least one condition previously specifiedby the first user would be satisfied. Subsequently, if the user requeststhat a light be turned on, or otherwise turns on a light within theirhome, another condition will be satisfied, and the automated assistantcan cause the playback of the 90s playlist to be rendered, at leastaccording to the aforementioned example.

In some implementations, conditions for modifying an ongoing operationcan be based on interactions between a user and an automated assistant,and/or interactions between a user and a device via which an automatedassistant is not directly accessible. For instance, the user can providea spoken utterance, such as, “Assistant, when I make coffee in themorning, read my schedule for the day.” A coffee maker within the homeof the user may not be a smart device, or otherwise connected to a localarea network. However, the automated assistant can determine that theuser is making coffee according to one or more situational conditionsand/or characteristics typically exhibited when the user is makingcoffee. For instance, in response to the user requesting that anoperation be contingent upon “making coffee,” the automated assistantcan be invoked and/or otherwise responsive to the sound of a percolator,an image of a cup or pot of coffee, a smell corresponding to coffee,and/or any other detectable environmental characteristic attributable toa process of making coffee. In this way, the user can give the automatedassistant permission to detect various happenings, characteristics,and/or tasks performed at the direction of the user, in order todetermine whether a particular requested condition has been satisfied.

In some implications, other data can be used in combination withdetermined situational characteristics in order to verify that a userwho has requested the creation of the condition is the one causing aparticular condition to be satisfied. For instance, with priorpermission from the user, voice data, facial recognition data, and/orany other user permitted data can be processed to verify which user issatisfying a particular condition. As an example, if the user hadprovided a spoken utterance prior to making coffee, the automatedassistant can process the spoken utterance to verify who provided thespoken utterance and therefore deduce that the same user made thecoffee. Therefore, any action (e.g., such as reading a schedule for theday) that depends on a particular user making coffee (or any othercondition), can be initialized when the user provides a spoken utterance(from which a voice characteristic can be determined) and subsequentlysatisfies a condition (e.g., making coffee, which can provideenvironmental changes, such as mechanical noise and changes intemperature).

The above description is provided as an overview of some implementationsof the present disclosure. Further description of those implementations,and other implementations, are described in more detail below.

In some implementations, a method implemented by one or more processorsis set forth as including operations such as receiving a user input thatcauses a computing device to generate a setting for controlling a wakealarm when any one of one or more conditions is satisfied, wherein thecomputing device provides access to an automated assistant and the userinput is received at an automated assistant interface of the computingdevice. The operations can further include accessing, when the computingdevice is operating, data that provides an indication of whether any oneof the one or more conditions has been satisfied, wherein the one ormore conditions include at least an event condition and a situationalcondition, and wherein the situational condition is satisfied when theuser has performed one or more interactions with the automated assistantwithin a threshold period of time of an event corresponding to the eventcondition, and the one or more interactions are selected to satisfy thesituational condition based on one or more other interactions, betweenthe user and the automated assistant, that previously occurred within aperiod of time of a past occurrence of the event. The operations canfurther include determining, based on accessing the data, whether thedata indicates that the any one of the one or more conditions has beensatisfied. The operations can further include, when the situationalcondition is determined to be satisfied prior to satisfaction of theevent condition: causing performance of the wake alarm to be dismissed,such that an output corresponding to the wake alarm is omitted when theevent subsequently occurs.

In some implementations, the event is an occurrence of a particular timeand performance of the wake alarm includes causing a sound to be emittedfrom the computing device or a separate computing device. In someimplementations, the operations can further include, when the eventcondition is determined to be satisfied by a temporal arrival of theevent: causing the wake alarm to be executed such that the output,corresponding to the wake alarm, is provided by the computing device ora separate computing device. In some implementations, the one or moreconditions include at least one condition further selected from a userinput condition, and the method further comprises: when the user inputcondition is determined to be satisfied prior to the event subsequentlyoccurring: causing the wake alarm to be dismissed, such that the output,corresponding to the wake alarm, is omitted when the event subsequentlyoccurs. In some implementations, the event condition includes at least atemporal parameter that identifies a particular time that, when atemporal arrival of the particular time occurs, causes the eventcondition to be satisfied. In some implementations, the automatedassistant is configured to perform a routine and cause multipledifferent actions to be initialized when performing the routine, andwherein the multiple different actions of the routine include causing aseparate computing device to perform another wake alarm that causes theseparate computing device to provide a different output, which isdirectly perceivable by a user. In some implementations, the operationscan further include when the situational condition is determined to besatisfied prior to the event that corresponds to the event condition:causing the automated assistant to initialize performance of theroutine, prior to the event, in response to determining that thesituational condition has been satisfied. In some implementationsdetermining whether the data indicates that that any one of the one ormore conditions has been satisfied includes determining that the dataindicates that a user has provided a spoken utterance, which has invokedthe automated assistant prior to the event. In some implementations,causing the automated assistant to initialize performance of the routineincludes causing the automated assistant to perform an action that isdifferent from a requested action specified by the user via the spokenutterance.

In other implementations, a method implemented by one or more processorsis set forth as including operations such as receiving a first userinput that causes a computing device to generate a setting forperforming an operation when an event occurs, wherein the computingdevice provides access to an automated assistant that is configured toperform a routine and cause multiple different actions to be initializedwhen performing the routine. The operations can further includereceiving a second user input that causes the automated assistant togenerate another setting for performing the routine when any one of oneor more conditions is satisfied, wherein the one or more conditionsinclude at least one condition selected from a user input condition, andthe user input condition is satisfied when the user has caused theoperation to be dismissed during a particular time selected from a timeprior to the event and another time after the event. The operations canfurther include accessing, subsequent to the first user input and thesecond user input, data that provides an indication of whether the anyone of the one or more conditions has been satisfied. The operations canfurther include determining, based on accessing the data, that the dataindicates that the any one of the one or more conditions has beensatisfied. The operations can further include, when the data indicatesthe user input condition has been satisfied at the time prior to theevent: causing performance of the operation to be dismissed, such thatan output, to be provided by the computing device and corresponding tothe operation, is omitted when the event subsequently occurs, andcausing the automated assistant to initialize performance of theroutine, prior to the event, in response to determining that the any oneof the one or more conditions has been satisfied.

In some implementations, the data indicates that the user inputcondition has been satisfied by a third user input for causing theoperation to be dismissed at the time prior to the event. In someimplementations, the third user input is a physical gesture, theoperation is the computing device providing an audible alarm output, andthe event is a temporal arrival of a particular clock time. In someimplementations, the operations can further include, when the dataindicates that the user input condition has been satisfied at the othertime after the event: causing the output, being provided by thecomputing device according to the setting, to be dismissed in responseto a third user input from the user, and causing, in response to thethird user input from the user, the automated assistant to initializeperformance of the routine at the other time after the event. In someimplementations, the one or more conditions include at least onecondition further selected from a situational condition, and the methodfurther comprises: when the data indicates that the situationalcondition has been satisfied at the time prior to the event: causingperformance of the operation to be dismissed, such that the output, tobe provided by the computing device and corresponding to the operation,is omitted when the event subsequently occurs, and causing the automatedassistant to initialize performance of the routine, prior to the event,in response to determining that the situational condition has beensatisfied. In some implementations, the situational condition includesat least an input parameter that identifies a particular user inputthat, when received at a separate computing device, causes the data tobe generated for indicating that the situational condition has beensatisfied. In some implementations, the particular user inputcorresponds to a spoken utterance, provided by the user, to an automatedassistant interface of the separate computing device, and the spokenutterance characterizes a request for the automated assistant to performa particular action that is omitted from the routine.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as receiving datathat characterizes a first operational status of a first computingdevice and a second operational status of a second computing device,wherein each of the first computing device and the second computingdevice provide access to an automated assistant that is configured to beresponsive to natural language inputs from a user and perform a routinethat includes initializing multiple different actions to be performed.The operations can further include determining, based on the data, thatthe first operational status indicates the first computing device isscheduled to perform an operation in response to at least an eventcondition being satisfied, and the second operational status indicatesthat the second computing device is to perform an action of an automatedassistant routine in response to a condition being satisfied, whereinthe condition is a particular condition selected from a user inputcondition, a situational condition, and the event condition. Theoperations can further include receiving, subsequent to receiving thedata, other data that characterizes a presence of the user within anenvironment that includes the first computing device and the secondcomputing device. The operations can further include determining, basedon the other data, whether the condition has been satisfied. Theoperations can further include, when the situational condition isdetermined, based on the other data, to have been satisfied prior to anevent that corresponds to the event condition: causing performance ofthe operation to be dismissed, such that an output, to be provided bythe first computing device and corresponding to the operation, isomitted when the event subsequently occurs, and causing the automatedassistant to initialize performance of the routine, prior to the event,in response to determining that the situational condition has beensatisfied.

In some implementations, the operations can further include, when theevent condition and the situational condition are determined, based onthe data, to be satisfied and the first computing device is providingthe output subsequent to the event: causing the first computing deviceto discontinue providing the output, and causing the automated assistantto initialize performance of the routine, prior to the event, inresponse to determining that the situational condition has beensatisfied. In some implementations, the event is a temporal arrival of aclock time and the situational condition is satisfied when, at least,the presence of the user, characterized by the other data, indicates theuser has changed locations within the environment. In someimplementations, the output is an audible output that is directlyperceivable by the user, and is provided by the first computing devicevia a speaker that is configured to provide a natural language output tothe user during performance of the routine by the automated assistant.In some implementations, the action of the automated assistant routineincludes causing a particular device selected from the first computingdevice and the second computing device to download additional data,which has been generated at a server device, and cause the particulardevice to provide a particular output, which is based on the additionaldata, from an automated assistant interface of the particular device. Insome implementations, the action of the automated assistant routineincludes causing a particular device, which is different from the firstcomputing device and the second computing device, to perform anotheroperation that is different from the operation that the first computingdevice is scheduled to perform.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as receiving afirst user input that causes a computing device to generate a settingfor performing an action when an event occurs, wherein the computingdevice provides access to an automated assistant and the first userinput is received at an automated assistant interface of the computingdevice. The operations can further include receiving, via the automatedassistant interface of the computing device, a second user input thatcauses the automated assistant to generate another setting for causing amodification of a performance of the action in response to any one ofone or more conditions being satisfied, wherein the one or moreconditions are specified by a user via the second user input. Theoperations can further include, when the event occurs subsequent toreceiving the first input and the second input: causing the action to beperformed in response to the event occurring, accessing, subsequent tocausing the action to be performed, data that provides an indication ofwhether the any one of the one or more conditions has been satisfied,and when the data indicates the any one of the one or more conditionswas satisfied after the event occurred: causing performance of theaction to be modified according to the setting.

In some implementations, the one or more conditions include: determiningthat a status of a device has been modified, and verifying that the userwho provided the first user input and/or the second user input alsoperformed the modification to the status of the device. In someimplementations, the data includes audio data and the method furthercomprises: when the event occurs subsequent to receiving the first inputand the second input: determining that the audio data corresponds to avoice signature of the user who provided the first user input and/or thesecond user input. In some implementations, the operations furtherinclude, when the event has not occurred but the any one of the one ormore conditions is satisfied prior to the event occurring: causingperformance of the action to be dismissed such that an output, to beprovided by the computing device in accordance with the action, isomitted when the event subsequently occurs.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by one or more processors(e.g., central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a methodsuch as one or more of the methods described above and/or elsewhereherein. Yet other implementations may include a system of one or morecomputers and/or one or more robots that include one or more processorsoperable to execute stored instructions to perform a method such as oneor more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a view of a user requesting that a setting begenerated at an assistant device.

FIG. 1B illustrates a view of a user causing an automated assistant toperform a routine in response to a gesture that is directed at affectingan ongoing operation being performed by a device.

FIG. 2A illustrates a view of a user causing a setting to be generatedfor invoking an automated assistant in response to the user affecting anoperation to be performed by a computing device.

FIG. 2B illustrates a view of the user causing the alarm to bepre-emptively dismissed based on one or more actions performed prior tothe alarm going off.

FIG. 3 illustrates a system for enabling a user to pre-emptively dismissa scheduled operation and/or cause an automated assistant to perform oneor more actions in response to a scheduled operation being pre-emptivelydismissed.

FIG. 4A and FIG. 4B illustrate methods for pre-emptively dismissing aforthcoming and/or anticipated scheduled operation when one or moreconditions are satisfied.

FIG. 5 is a block diagram of an example computer system.

DETAILED DESCRIPTION

FIG. 1A illustrates a view 100 of a user 118 requesting that a settingbe generated at an assistant device 110. Specifically, the user 118 canprovide a spoken utterance 106 when they are in their room 108, in orderto generate a setting that will affect a function of an automatedassistant. The generation of the setting can be requested by the user118 in order to indicate a dependency of execution of an automatedassistant routine on one or more particular conditions, operations,occurrences, and/or any other characteristic or action that performanceof a routine can be based on. For example, the user 118 can provide aspoken utterance 106 such as, “Assistant, set an alarm for 6:30 A.M.,and start my morning routine when I dismiss my alarm.” The spokenutterance 106 can be received at an assistant device 110, such as astandalone speaker device 102, which includes an assistant interface114, such as a microphone, speaker, and/or any other apparatus capableof receiving an input and/or providing output. In response to theassistant device 110 receiving the spoken utterance 106, the assistantdevice 110 can provide a natural language output 104 such as, “Ok,”thereby confirming receipt of the spoken utterance 106.

The spoken utterance 106 can be processed at the assistant device 110and/or a remote device, such as a server device, in order to generateclient data 116. The client data 116 can characterize a settinggenerated in response to the client automated assistant 112 receiving arequest that is based on the spoken utterance 106. Specifically, theclient data 116 can characterize a setting, which is capable of causingan alarm to go off at 6:30 A.M., and the automated assistant to performthe “morning” routine when the alarm is dismissed by the user 118. Insome implementations, in response to receiving the spoken utterance 106,the automated assistant can determine multiple different conditions,which when satisfied, cause the particular operations defined by thespoken utterance 106 to be performed. For example, the client data 116can characterize a condition that includes a parameter characterizing auser input for dismissing the alarm, therefore when such a user input isreceived, the condition can be satisfied. The client data 116 can alsocharacterize as a condition that includes another parametercharacterizing an event input for causing an alarm operation toinitialize. The other parameter can characterize the event input as atemporal arrival of a clock time, therefore, when the clock time occurs,the alarm can be emitted by the assistant device 110 and/or any otherdevice associated with the user 118.

FIG. 1B illustrates a view 120 of a user 118 causing an automatedassistant to perform a routine in response to a gesture that is directedat affecting an ongoing operation being performed by a device.Specifically, FIG. 1B illustrates a home of the user 118 that includesmultiple different devices. The assistant device 110 can be located inthe room 108, and other devices can be located outside of the room 108.For instance, the user 118 can have another assistant device 122, suchas a standalone speaker device 130. Additionally, the user 118 can haveother client devices connected with the assistant devices over a localarea network. The other client devices can include, for example, acoffee maker 132 and/or a thermostat 134.

According to FIG. 1A, the user has caused a setting to be created for analarm to go off at a particular time, and an automated assistant routineto be performed when the alarm is dismissed. When the alarm subsequentlygoes off, as illustrated in FIG. 1B, the alarm can be dismissed by theuser 118. For instance, as illustrated in FIG. 1B, the assistant device110 can be providing an output 144, which can include emitting an alarm.When the alarm is being emitted by the assistant device 110 in the room108, the user 118 can perform a gesture to indicate a willingness orintention to dismiss the alarm. The gesture for dismissing the alarm canbe, for example, a spoken gesture, a gesture performed via motion of anappendage, and/or any other gesture that can be an input to a computingdevice. In response to the alarm being dismissed by the gesture from theuser 118, the automated assistant can initialize performance of the“morning” routine. The initialization of this routine can be based onthe setting generated according to FIG. 1A.

The routine to be performed by the automated assistant can becharacterized by client data 116, client data 128, and/or any other datathat can be accessible to the automated assistant. The client data 128can characterize one or more actions to be performed when the routine isbeing executed by the automated assistant. For instance, a clientautomated assistant 124, accessible via the assistant device 122, canprovide the morning news when the routine is being executed.Furthermore, as part of the “morning” routine, the client automatedassistant 124 can cause the coffee maker 132 to perform an operation 140such as starting to make coffee, and cause the thermostat 134 to performan operation 142 such as stopping motion detection (e.g., as part of asecurity mode of thermostat 134). In other words, as a result of thesetting created at the request of the user 118, when the user 118dismissed the alarm being emitted by the assistant device 110, theclient automated assistant 124 can initialize performance of the“morning” routine.

Causing the automated assistant to perform a routine in this way canreduce a number of user inputs that need to be processed at one or moredevices. For example, instead of the user 118 providing a spokenutterance to an assistant interface 126 of the assistant device 122 inorder to initialize the “morning” routine, the user 118 can cause the“morning” routine to be performed in response to dismissing the alarm.In some implementations, the initialization of the routine by theautomated assistant can be contingent upon who dismissed the alarm. Inother words, if someone other than the user 118 dismissed the alarm, theautomated assistant can bypass initializing the routine according to thesetting. In this way, should the user 118 be traveling or otherwise nothome when the alarm is going off, and the alarm is subsequentlydismissed by another user, the automated assistant will omit executingthe “morning” routine. This can preserve energy and computationalresources that might otherwise be expended upon performing the routine.Furthermore, this can provide an additional level of security for theuser and/or other users within the home. Voice identification can beused to determine whether the user 118 or a different user is dismissingthe alarm. Alternatively, or additionally, various environmental dataand/or user data can be used to determine whether the user 118 is theperson dismissing the alarm, or whether the alarm is being dismissed bysomeone other than the user 118.

FIG. 2A illustrates a view 200 of a user 218 causing a setting to begenerated for invoking an automated assistant in response to the useraffecting an operation to be performed by a computing device.Specifically, the user 218 can request that an automated assistantperform a routine in response to the user affecting another operationperformed by the computing device. In response to the request from theuser 218, the automated assistant can determine multiple differentconditions that can be satisfied in order for the automated assistant toperform the routine. For example, when the user 218 is in their room 208with an assistant device 210, such as a standalone speaker device 202,the user 218 can provide a spoken utterance 206. The spoken utterance206 can be, for example, “Assistant, set an alarm for 6:30 A.M., andstart my morning routine when I dismiss my alarm.” The spoken utterance206 can be received at an assistant interface 214 of the assistantdevice 210. A client automated assistant 212 that is available at thestandalone speaker device 202 can generate client data 216 in responseto receiving the spoken utterance 206. The client data 216 cancharacterize one or more conditions that, when satisfied, cause theautomated assistant to perform the routine without the user necessarilyproviding a spoken utterance that identifies the routine.

For example, in response to receiving the spoken utterance 206, theclient automated assistant 212 can cause client data 216 to be generatedfor characterizing a user input condition. The user input condition caninclude a parameter of receiving a user input for dismissing the alarm.When the user input for dismissing the alarm is received, the user inputcondition can be considered satisfied, and the automated assistant caninitialize performance of the morning routine in response. The userinput condition can correspond to a user input that expresses awillingness or an intention of the user to stop the alarm when it isbeing emitted, or stopping the alarm from being subsequently emitted.For example, prior to a time that the alarm is set to go off, but aftera time that the user has been sleeping for a threshold period of time,the user can provide an input indicating that they would like theimpending alarm to be dismissed despite the alarm having not gone offyet.

Alternatively, or additionally, in response to receiving the spokenutterance 206, the client automated assistant 212 can generate clientdata 216 for characterizing an event condition. The event condition caninclude one or more parameters for an occurrence of an event. Forinstance, a temporal arrival of a particular clock time can be anoccurrence of an event that satisfies the event condition.Alternatively, the event can be one or more other time dependentactivities that can be acknowledged by a computing device. When theevent condition is satisfied, the alarm can be emitted by the standalonespeaker device 202 in order to wake the user. Furthermore, according tothe setting requested by the user, when the event condition issatisfied, the automated assistant can initialize performance of a“morning” routine. In this way, the automated assistant can initializeperformance of a routine without necessarily requiring the user toprovide another spoken utterance subsequent to the spoken utterance 206.

Additionally, or alternatively, in response to receiving the spokenutterance 206, the client automated assistant 212 can generate clientdata 216 for characterizing a situational condition. A situationalcondition can include one or more parameters for characterizing one ormore particular situations. The automated assistant can determine that aparticular situation is apparent or ongoing based on data from one ormore devices. For example, a situational condition can include one ormore parameters for characterizing a situation in which the user ismoving about their home outside of the room 208. The parameters caninclude a threshold level of audio data collected by a device outside ofthe room 208, a threshold level of motion detected by a device outsideof the room 208 and/or inside of the room 208, a threshold amount ofinteraction between the user and one or more devices, and/or any otherinformation that can indicate a particular situation that is apparent orongoing. When the situational condition is satisfied prior to a clocktime that is set for the alarm, the alarm can be preemptively dismissedand the automated system can initialize performance of the “morning”routine. When the situational condition is satisfied at, or after, atime when the alarm is set to go off, an output corresponding to thealarm can be dismissed, and the automated assistant can initializeperformance of the morning routine.

FIG. 2B illustrates a view 220 of the user 218 causing the alarm to bepre-emptively dismissed based on one or more actions performed prior tothe alarm going off. For example, a setting 244 of the assistant device210 can operate to effectively cause an alarm to go off in 23 minutesfrom a current time period. Regardless, the user 218 can cause the alarmto be pre-emptively dismissed by causing a situational condition to besatisfied. The situational condition can include parameters for the userinvoking the automated assistant, the user moving outside of the room208, and/or the user otherwise interacting with a device within theirhome, and/or outside of their home, prior to the scheduled time that thealarm is to go off. For instance, FIG. 2B illustrates the user 218providing, to an assistant device 222, a query such as, “What is theweather?” The query provided by the user 218 can be provided before atime that the alarm is scheduled to go off, thereby indicating to theassistant device 222 that the user is awake prior to the alarm goingoff. In some implementations, motion can be detected by a particulardevice (e.g., a thermostat 234) within the home of the user prior to atime when the alarm is configured to go off. Such detection incombination with one or more other detected activities can indicate thatthe user is awake prior to the alarm going off, or is otherwiseinterested in the alarm being dismissed prior to the alarm going off.

In response to the alarm being preemptively dismissed, a clientautomated assistant 224 can be notified that the alarm has beenpre-emptively dismissed. Furthermore, in response to the alarm beingpre-emptively dismissed, the client automated assistant 224 caninitialize performance of the “morning” routine. Client data 228 can bestored at the client assistant device 224, and/or a separate device suchas server device, and can characterize one or more actions to beperformed according to the routine. For example, the morning routine caninclude an action of causing the client automated assistant 224 toprovide the morning news, an action 240 of initializing a coffee maker232 to make coffee, and an action 242 of causing a thermostat 234 toexit a security mode and stop motion detection.

In some implementations, when the situational condition is satisfied bythe user 218 providing a query or command to the client automatedassistant 224, an action to be performed according to the routine can bemodified according to the query or command. For example, when thesituational condition is satisfied by the user 218 providing the query,“What is the weather?” the subsequent routine action of providing themorning news can be modified to include providing the weather. As aresult, an assistant interface 226 of the assistant device 222 canprovide a natural language output 236 such as, “Here is the weather andthe morning news . . . ” In this way, a “domino effect” can occur, atleast with respect to various device operations, in response to the userinteracting with the automated assistant prior to a time at which thealarm is configured to go off. Such an effect can reduce a number ofinputs that the user needs to provide in order to cause the automatedassistant to perform the same amount of activity. Furthermore, reducinga number of such inputs, such as spoken utterances to be received at anassistant interface 226, can preserve computational resources andnetwork resources.

When processing natural language inputs, computational resources can beexpended on converting speech to text, and parsing text for determiningcontent of the text. Furthermore, processing natural language inputs canoften require audio data and/or other data to be transmitted over anetwork for additional processing, there by consuming bandwidth.Therefore, by inferring certain user-invoked activity as a willingnessor intention for particular operations of a device to be initialized ordismissed, such resources can be preserved at a time when predictabilityof the user intentions may not be particularly high. Specifically,pre-emptively dismissing an alarm according to one or more useractivities can be a reasonable indicator that the user would prefer anyactions or routines contingent upon the alarm to be performed, despitethe alarm not actually going off at a set time.

FIG. 3 illustrates a system 300 for enabling a user to pre-emptivelydismiss a scheduled operation and/or cause an automated assistant 306 toperform one or more actions in response to a scheduled operation beingpre-emptively dismissed. The automated assistant 306 can operate as partof an assistant application that is provided at one or more computingdevices, such as a first client device 326, a second client device 334,and/or a remote computing device 302, such as a server computing device304. A user can interact with the automated assistant 330 via one ormore assistant interfaces 328, which can include one or more of amicrophone, a camera, a touch screen display, a user interface, and/orany other apparatus capable of providing an interface between a user andan application. For instance, a user can initialize the automatedassistant 330 by providing a verbal, textual, or a graphical input tothe assistant interface to cause the automated assistant 330 to performa function (e.g., provide data, control a peripheral device, access anagent, etc.). The first client device 326 can include a display device,which can be a display panel that includes a touch interface forreceiving touch inputs and/or gestures for allowing a user to controlapplications of the first client device 326 via the touch interface. Insome implementations, second client device 334 can lack a displaydevice, thereby providing an audible user interface output, withoutproviding a graphical user interface output. Furthermore, the secondclient device 334 can provide a user interface, such as a microphone,for receiving spoken natural language inputs from a user.

The first client device 326 and the second client device 334 can be incommunication with the remote computing device 302 over a network, suchas the interne. The first client device 326 and the second client device334 can offload computational tasks to the remote computing device 302in order to preserve computational resources at each of the first clientdevice 326 and the second client device 334. For instance, the remotecomputing device 302 can host an automated assistant 306, and the firstclient device 326 and the second client device 334 can transmit inputsreceived at one or more assistant interfaces to the remote computingdevice 302. However, in some implementations, the automated assistant306 can be hosted at the first client device 326 or the second clientdevice 334. In various implementations, all or less than all aspects ofthe automated assistant 306 can be implemented on the server computingdevice 304, the first client device 326, and/or the second client device334. In some of those implementations, aspects of the automatedassistant 306 are implemented via a local assistant application of thefirst client device 326 or the second client device 334 and interfacewith the remote computing device 302, which can implement other aspectsof the automated assistant 306. The remote computing device 302 canoptionally serve a plurality of users and their associated assistantapplications via multiple threads. In implementations where all or lessthan all aspects of the automated assistant 306 are implemented via alocal assistant application of the first client device 326 or the secondclient device 334, the local assistant application can be an applicationthat is separate from an operating system of the first client device 326or the second client device 334 (e.g., installed “on top” of theoperating system)—or can alternatively be implemented directly by theoperating system of the first client device 326 or the second clientdevice 334 (e.g., considered an application of, but integral with, theoperating system).

In some implementations, the automated assistant 306, automatedassistant 330, and/or automated assistant 336 can include an inputprocessing engine 308, which can employ multiple different modules forprocessing inputs and/or outputs for the first client device 326 and/orthe second client device 334. For instance, the input processing engine308 can include a speech processing module 310 that can process audiodata received at an assistant interface 328 and/or an assistantinterface 336 to identify the text embodied in the audio data. The audiodata can be transmitted from, for example, the first client device 326to the server computing device 304 in order to preserve computationalresources at the first client device 326.

The process for converting the audio data to text can include a speechrecognition algorithm, which can employ neural networks, word2vecalgorithms, and/or statistical models for identifying groups of audiodata corresponding to words or phrases. The text converted from theaudio data can parsed by a data parsing module 312 and made available tothe automated assistant as textual data that can be used to generateand/or identify command phrases from the user. In some implementations,output data provided by the data parsing module 312 can be provided to aparameter module 314 to determine whether the user has provided an inputthat corresponds to a particular action and/or routine capable of beingperformed by the automated assistant and/or an application or agent thatis capable of being accessed by the automated assistant. For example,assistant data 324 can be stored at the server computing device 304and/or the first client device 326, as client data 332, and can includedata that defines one or more actions capable of being performed by theautomated assistant 306, as well as parameters necessary to perform theactions.

When the input processing engine 308 has determined that a user hasrequested a particular action or routine be performed, the parametermodule 314 can determine one or more parameters for the particularaction or routine, and an output generating engine 316 can then providean output to the user based on the particular action, routine, and/orthe one or more parameters. For instance, in some implementations, inresponse to a user input, such as a gesture directed at the first clientdevice 326, the automated assistant 330 can cause data, whichcharacterizes the gesture, to be transmitted to the server computingdevice 304 for determining the action that the user is intending theautomated assistant 330 and/or automated assistant 306 to perform.

In some implementations, the automated assistant 330, the first clientdevice 326, the second client device 334, and/or the server computingdevice 304 can be responsive to one or more different types of gesturesdirected at the first client device 326. For instance, when the firstclient device 326 includes a speaker, a type of gesture that can be usedto control the volume can be a two-dimensional gesture (e.g., swiping atouch screen display or otherwise moving an appendage of the user in atleast two-dimensions with or without directly contacting a computingdevice) or a three-dimensional gesture (e.g., rotating two fingers onthe touch screen display or otherwise moving an appendage of the user inat least three-dimensions with or without directly contacting acomputing device).

In some implementations, a user can request that the automated assistant330 perform a routine when the automated assistant 330 determines thatthe user is awake. For instance, the user can provide a spoken utteranceto an assistant interface 328, such as, “Assistant, when I wake up,please perform my work routine.” The first client device 326 can receivethe spoken utterance and convert the spoken utterance to client data332, which can be transmitted to the server computing device 304 forfurther processing. An input processing engine 308 of the servercomputing device 304 can parameterize the received client data anddetermine a setting to be generated in response to receiving the clientdata. For instance, a routine engine 322 can identify a routine that theuser is specifically requesting be performed when the automatedassistant determines that the user is awake. The routine can beidentified using data generated from audio data that is based on thespoken utterance. When the routine has been identified, the servercomputing device 304 can generate a setting that will cause the workroutine to be performed when the automated assistant 330 determines thatthe user is awake. The setting can be stored as assistant data 324 atthe server computing device 304, and can be stored locally at the firstclient device 326 and/or the second client device 334.

The “work routine” can include one or more different actions to beperformed by the automated assistant 306, the automated assistant 330,the automated assistant 338, and/or any other application or agentcapable of interacting with a computing device. For example, a workroutine can include an action of initializing a display panel 344 of aninternet of things (IoT) device 342 to graphically present morning news,and an action of causing the second client device 334 to audibly presenta current day's schedule for the user from an assistant interface 336,such as an audio speaker.

In order to effectively employ the setting for invoking the “workroutine” in response to the user waking up, the automated assistantshould also be able to determine whether the user is awake, at least toa particular degree. Determining whether the user is awake can be basedon data that characterizes device statuses 318, such as operatingstatuses for the first client device 326, the second client device 334,the IoT device 342, the server computing device 304, and/or any otherdevice or module capable of interacting the user. For instance, a changein device status from emitting an alarm to having the alarm dismissed bythe user can be an indication that the user is awake. Alternatively, oradditionally, a change from a sleep state for the second client device334 to an on state, and a user providing a spoken utterance, can bothindicate that the user is awake, if the user was determined to be sleepfor a threshold period of time immediately prior to the awakeindications. Alternatively, or additionally, device statuses 318 andcontextual data 320 can be used to determine whether a user is awake.For example, both a change in device status and an indication, from thecontextual data 320, that the user is moving about their home, canindicate to the automated assistant that the user is awake. In responseto determining the user is awake, the automated assistant can initializeperformance of the work routine.

In some implementations, device statuses 318 and/or contextual data 320,accessed prior to a scheduled operation, such as an alarm, can indicatethat the user is awake and, in response, cause the automated assistantto perform the work routine. For example, prior to a time when the alarmis scheduled to go off, an IoT device 342 can detect the presence of theuser and the second client device 334 can receive a spoken utterancefrom the user, such as, “Assistant, turn on the lights.” In response todetecting the presence of the user, receiving the spoken utterance,and/or turning on the lights, the server computing device 304 and/or thefirst client device 326 can determine that the user is awake. In someimplementations, in response to determining that the user is awake priorto a scheduled operation, the automated assistant can cause a scheduledoperation be performed prior to an originally scheduled time, and/orcause the scheduled operation to be bypassed at the scheduled time.Furthermore, in response to determining that the user is awake, andaccording to a setting requested by the user, the automated assistantcan initialize performance of one or more actions or one or moreroutines. In this way, when the scheduled operation is an alarm, theuser can initialize performance of the routine and dismiss their alarmwith less inputs than would otherwise be required. This can reduce anumber of inputs that need to be processed by one or more devices ofFIG. 3, and can preserve network bandwidth, as less audio datacorresponding to spoken utterances would need to be transmitted betweendevices for processing.

In some implementations, a determination as to whether a scheduledoperation should be dismissed and/or adjusted in time of/forperformance, can be based on learned behavior of the user and/or anyother person that can interact with the automated assistant. Forexample, a user may interact with the first client device 326 in orderto reset their alarm just before it was scheduled to go off. Althoughthe user would likely be awake under the circumstances, it may be awaste of resources to initialize performance of an automated assistantroutine under such circumstances, since the user may be adjusting theiralarm in order to sleep longer. Therefore, in order to determine whetherthe user is awake, and therefore willing to have the routine beperformed, contextual data 320 and/or device statuses 318 can beaccessed to determine whether the user is awake and would prefer theroutine be initialized.

For example, data accessible to the automated assistant can indicatethat, after the user's alarm goes off, the user typically requests theautomated assistant 330 to turn on the lights in their kitchen and theuser also typically turns on their IoT device 342. Therefore, theautomated assistant can cause assistant data 324 to be generated forcharacterizing such interactions as an indication that the user isawake. As a result, should the user perform such actions prior to, orduring, their alarm going off, such interactions can cause the alarm tobe dismissed (or pre-emptively dismissed) and cause the automatedassistant routine to be initialized. In some implementations, acondition engine 346 can process assistant data 324, contextual data320, device statuses 318, and/or any other data to determine whether oneor more conditions have been satisfied for affecting a scheduledoperation and/or one more actions to be performed by an automatedassistant. The condition engine 346 can process such data to determineparameters for satisfying particular conditions, such as, but notlimited to, a user input condition, a situational condition, and/or anevent condition, as discussed herein. When one or more conditions,corresponding to a particular setting requested by the user, aresatisfied, one or more actions, routines, and/or operations associatedwith the particular setting can be affected by satisfaction of the oneor more conditions.

FIG. 4A and FIG. 4B illustrate a method 400 and a method 414 forpre-emptively dismissing a forthcoming and/or anticipated scheduledoperation when one or more conditions are satisfied. The method 400 andthe method 414 can each be performed by one or more computing devices,applications, and/or any other apparatus or module capable of storing oraccessing a schedule. The method 400 can include an operation 402 ofdetermining that a user has provided a first user input requesting thata setting be generated for performing an operation when one or moreconditions are satisfied. For example, the user can provide a firstinput via an automated assistant for establishing an alarm that will gooff the following morning. The first input can be, for example, a spokenutterance such as, “Assistant, please set an alarm for 6 A.M. tomorrowmorning.” The spoken utterance can be received at an automated assistantinterface of a computing device, such as a microphone of the computingdevice. In response to the computing device receiving the spokenutterance, audio data can be generated for characterizing the spokenutterance. The audio data can be processed at the computing deviceand/or transmitted to a separate computing device for processing. Basedon the processing, the computing device can establish a setting forcausing an audible sound to be emitted from a speaker of the computingdevice when a temporal arrival of the clock time (e.g., 6 A.M.) occursthe following morning. The temporal arrival of the clock time can becharacterized as an event condition that, when satisfied, causes thealarm sound to be emitted by the computing device.

In some implementations, in response to the user requesting the settingbe generated, the computing device and/or the automated assistant canestablish one or more other conditions that, when satisfied, cause thecomputing device to perform or bypass the requested operation. Forexample, the automated assistant can cause a user input condition to beestablished such that, when one or more user inputs are received priorto an event, corresponding to the event condition, the requestedoperation can be performed or bypassed when the event occurs.Additionally, or alternatively, the computing device and/or theautomated assistant can establish yet another one or more conditionsthat, when satisfied, cause the computing device to perform or bypassthe requested operation. For example, the automated assistant can causea situational condition to be established such that, when dataaccessible to the automated assistant indicates that a particularsituation and/or context is apparent, prior to an event corresponding tothe event condition, the requested operation can be performed orbypassed when the event occurs. In some implementations, a user input,such as a spoken utterance to the automated assistant, can satisfy theuser input condition. Alternatively, or additionally, a situation or acontext that can satisfy the situational condition, can include the userturning on a light within their home.

The method 400 can further include an optional operation 404 ofdetermining that the user has provided a second user input requestingthat another setting be generated. Specifically, the second user inputcan correspond to a request for causing the automated assistant toperform a routine when one or more conditions are satisfied. Forinstance, the user can provide a spoken utterance such as, “Assistant,when I dismiss my alarm, please perform my ‘prepare for work’ routine.”In response to receiving the spoken utterance, the computing deviceand/or the automated assistant can generate a setting for performing the‘prepare for work’ routine when one or more conditions are satisfied.The one or more conditions can include at least one condition selectedfrom a user input condition, a situational condition, and/or an eventcondition. In some implementations, a situational condition can belearned over time, by the automated assistant and with permission fromthe user, as the user takes particular actions associated with therequested setting. For example, despite the user requesting that the‘prepare for work’ routine be performed when the user dismisses theiralarm, the routine can nonetheless be performed when they performcertain actions that indicate a willingness or interest in dismissingtheir alarm. For example, initially a user may not have a coffee machinein their home when they have made the aforementioned request. However,should the user purchase a coffee machine and use the coffee machine tomake coffee each morning before work, the automated assistant canacknowledge such behavior. Furthermore, the automated assistant cangenerate data that characterizes the behavior as an indication that theuser is awake and/or otherwise expressing a willingness to dismiss theiralarm. Therefore, subsequent to the data being generated by theautomated assistant, when the user makes coffee in the morning prior toa time that is set for their alarm, such an action can be acknowledgedby the automated assistant and satisfy a situational condition fordismissing the alarm and, optionally, invoking the automated assistantto perform the ‘prepare for work’ routine.

The user input condition, which can be established for invoking theautomated assistant to perform the routine, can be, for example, anaction of the user explicitly dismissing the alarm when the alarm isgoing off. The event condition, which can be established for invokingthe automated assistant to perform the routine, can be, for example, atemporal arrival of the clock time corresponding to a time at which theuser requested that the alarm go off In this way, duplicative userinputs do not necessarily need to be provided by the user in order tocause various actions to be performed by the automated assistant, and/orany other computing device.

The method 400 can further include an operation 406 of accessing datathat provides an indication of whether the one or more conditions havebeen satisfied. The data can be generated and/or provided by thecomputing device, the automated assistant, a server device that is incommunication with the computing device and/or at the automatedassistant, and/or any other application or device capable of interactingwith the automated assistant. For example, the data can be provided byone or more Internet of Things (IoT) devices, such as an appliance thatis connected to the internet. Therefore, the data can characterize oneor more interactions between the user and the appliance. For example,when the appliance is a copy machine in a kitchen of a home of the user,the data can indicate whether the user has caused a performance of acoffee making process at the coffee machine.

The method 400 can further include an operation 408 of determining,based on the data, whether the event has occurred. The event can be anyoccurrence corresponding to, or identified by, data that is associatedwith the event condition. Specifically, the event condition can bestored as data that includes a parameter identifying how the eventcondition can be satisfied. When the event condition corresponds to analarm, the parameter can identify a clock time at which the alarm is setto go off. Therefore, when the data indicates that a temporal arrival ofthe clock time has occurred, the method 400 can proceed to continuationelement “A,” which indicates that the method 400 proceeds fromcontinuation element “A” of FIG. 4A and continues to method 414 atcontinue element “A” at FIG. 4B.

When the data indicates that the event has not occurred, the method 400can proceed to operation 410, which can include a determination ofwhether a user input condition has been satisfied. A user inputcondition can refer to a direct or indirect user input being received atone or more particular devices. For example, a user input condition canbe defined by data that characterizes one or more user inputs that, whenprovided by a user prior to the event occurring, will satisfy the userinput condition. When the operation to be performed is an alarm that isset by the user, a user input condition can be the user specificallyrequesting that the alarm be dismissed prior to the time at which thealarm was supposed to go off.

Alternatively, or additionally, the user input condition can correspondto the user interacting with the automated assistant. For example, ifthe user interacts with the automated assistant within a thresholdperiod of time before the alarm supposed to go off, the user inputcondition can be considered satisfied. For instance, the user canprovide a spoken utterance to a particular device for invoking theautomated assistant to cause a web search to be performed at thedirection of the automated assistant. If this occurs prior to the eventoccurring, this interaction can indicate that the user is awake prior totheir alarm going off, and it can therefore be assumed by the automatedassistant that the alarm is no longer necessary. When the user inputcondition is satisfied, the method 400 can proceed to the method 414 viacontinuation element “B.” Specifically, continuation element “B” canrepresent a continuation point between the method 400 and the method414, which also includes the continuation element “B.”

When the user input condition has not been satisfied at operation 410,the method 400 can proceed to operation 412. The operation 412 caninclude determining whether a situational condition has been satisfied.A situational condition can be satisfied in response to one or moredevices detecting an activity of the user, with permission from theuser. In some implementations, the situational condition can besatisfied when a motion of the user is detected, thereby indicating thatthe user is awake or in motion prior to the event occurring.Additionally, or alternatively, the situational condition can besatisfied when an input from the user has been received at one or moredifferent devices, such as when the user turns on a light, adjusts atemperature of their home, is detected by one or more sensors (e.g., amicrophone or proximity sensor) in the home, and/or when any otherchange in a situation associated with the user has occurred. When thesituational condition has been satisfied, the method 400 can proceed tothe method 414 via a continuation element “C,” which is illustrated inFIG. 4A and FIG. 4B. When the situational condition has not beensatisfied, the method 400 can return to the operation 406 where furtherdata can be accessed to determine whether one or more conditions havebeen satisfied for performing the operation previously discussed withrespect to the operation 402.

The method 414 can proceed from the continuation element “A” to anoperation 416, which can include causing performance of the operation,such that an output from the computer device is provided in response tothe event occurring. When the event refers to a temporal arrival of atime, an output of the computing device can be provided when thatparticular time occurs. When the operation refers to an alarm having anaudio and/or visual output, the audio and/or visual output can beprovided when the temporal arrival of the alarm time occurs.

The method 414 can proceed from continuation element “B” andcontinuation element “C” to an operation 418, which can include causingperformance of the operation to be dismissed, such that an output fromthe computing device is omitted when the event subsequently occurs. Whenthe operation refers to an alarm, the output can refer to an audioand/or visual output that is provided by the computing device.Therefore, when the performance of the operation is dismissed, thecomputing device would not provide the audio and/or visual output fromthe computing device when the event occurs, but rather, the computingdevice can bypass providing such output in view of the user indicatingthat they are awake prior to their alarm going off.

The method 414 can optionally include an operation 420 of causing theautomated assistant to initialize performance of a routine. Theoperation 420 can optionally follow the operation 416 or the operation418, depending on whether one or more particular conditions have beensatisfied. Causing the automated assistant to perform the routine inresponse to either the operation (e.g., the alarm) being performed, orthe operation being dismissed, can provide a benefit of decreasing anumber of user inputs that need to be processed over a lifetime of oneor more computing devices. For instance, some amount of speechprocessing would necessarily need to be performed if the user wasrequired to provide a spoken utterance to an automated assistantinterface every time the user wanted the automated assistant to performthe routine. Furthermore, network bandwidth can be preserved, as theuser input would not necessarily need to be transmitted to a remoteserver for processing. Rather, processing of the user input can bebypassed in view of a willingness of the user to cause the operation(e.g., the alarm) to be dismissed before a particular event occurs.

FIG. 5 is a block diagram of an example computer system 510. Computersystem 510 typically includes at least one processor 514 whichcommunicates with a number of peripheral devices via bus subsystem 512.These peripheral devices may include a storage subsystem 524, including,for example, a memory 525 and a file storage subsystem 526, userinterface output devices 520, user interface input devices 522, and anetwork interface subsystem 516. The input and output devices allow userinteraction with computer system 510. Network interface subsystem 516provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 522 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 510 or onto a communication network.

User interface output devices 520 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 510 to the user or to another machine or computersystem.

Storage subsystem 524 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 524 may include the logic toperform selected aspects of method 400, method 414, and/or to implementone or more of an automated assistant, automated assistant 110,standalone speaker 102, assistant device 110, client automated assistant124, assistant device 210, client automated assistant 212, assistantdevice 222, client automated assistant 224, server computing device 304,first client device 326, second client device 334, IoT device 342,and/or any other device, apparatus, module, operation, and/orapplication discussed herein.

These software modules are generally executed by processor 514 alone orin combination with other processors. Memory 525 used in the storagesubsystem 524 can include a number of memories including a main randomaccess memory (RAM) 530 for storage of instructions and data duringprogram execution and a read only memory (ROM) 532 in which fixedinstructions are stored. A file storage subsystem 526 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 526 in the storage subsystem 524, or inother machines accessible by the processor(s) 514.

Bus subsystem 512 provides a mechanism for letting the variouscomponents and subsystems of computer system 510 communicate with eachother as intended. Although bus subsystem 512 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 510 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 510depicted in FIG. 5 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 510 are possible having more or fewer components thanthe computer system depicted in FIG. 5.

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore it is stored or used, so that personal identifiable informationis removed. For example, a user's identity may be treated so that nopersonal identifiable information can be determined for the user, or auser's geographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

1. A method implemented by one or more processors, the methodcomprising: receiving a user input that causes a computing device togenerate a setting for controlling a wake alarm when any one of one ormore conditions is satisfied, wherein the computing device providesaccess to an automated assistant and the user input is received at anautomated assistant interface of the computing device; accessing, whenthe computing device is operating, data that provides an indication ofwhether any one of the one or more conditions has been satisfied,wherein the one or more conditions include at least an event conditionand a situational condition, and wherein the situational condition issatisfied when the user has performed one or more interactions with theautomated assistant within a threshold period of time of an eventcorresponding to the event condition, and the one or more interactionsare selected to satisfy the situational condition based on one or moreother interactions, between the user and the automated assistant, thatpreviously occurred within a period of time of a past occurrence of theevent; determining, based on accessing the data, whether the dataindicates that the any one of the one or more conditions has beensatisfied; and when the situational condition is determined to besatisfied prior to satisfaction of the event condition: causingperformance of the wake alarm to be dismissed, such that an outputcorresponding to the wake alarm is omitted when the event subsequentlyoccurs.
 2. The method of claim 1, wherein the event is an occurrence ofa particular time and performance of the wake alarm includes causing asound to be emitted from the computing device or a separate computingdevice.
 3. The method of claim 1, further comprising: when the eventcondition is determined to be satisfied by a temporal arrival of theevent: causing the wake alarm to be executed such that the output,corresponding to the wake alarm, is provided by the computing device ora separate computing device.
 4. The method of claim 3, wherein the oneor more conditions include at least one condition further selected froma user input condition, and the method further comprises: when the userinput condition is determined to be satisfied prior to the eventsubsequently occurring: causing the wake alarm to be dismissed, suchthat the output, corresponding to the wake alarm, is omitted when theevent subsequently occurs.
 5. The method of claim 1, wherein the eventcondition includes at least a temporal parameter that identifies aparticular time that, when a temporal arrival of the particular timeoccurs, causes the event condition to be satisfied.
 6. The method ofclaim 1, wherein the automated assistant is configured to perform aroutine and cause multiple different actions to be initialized whenperforming the routine, and wherein the multiple different actions ofthe routine include causing a separate computing device to performanother wake alarm that causes the separate computing device to providea different output, which is directly perceivable by a user.
 7. Themethod of claim 6, further comprising: when the situational condition isdetermined to be satisfied prior to the event that corresponds to theevent condition: causing the automated assistant to initializeperformance of the routine, prior to the event, in response todetermining that the situational condition has been satisfied.
 8. Themethod of claim 7, wherein determining whether the data indicates thatthat any one of the one or more conditions has been satisfied includesdetermining that the data indicates that a user has provided a spokenutterance, which has invoked the automated assistant prior to the event.9. The method of claim 8, wherein causing the automated assistant toinitialize performance of the routine includes causing the automatedassistant to perform an action that is different from a requested actionspecified by the user via the spoken utterance.
 10. A method implementedby one or more processors, the method comprising: receiving a first userinput that causes a computing device to generate a setting forperforming an operation when an event occurs, wherein the computingdevice provides access to an automated assistant that is configured toperform a routine and cause multiple different actions to be initializedwhen performing the routine; receiving a second user input that causesthe automated assistant to generate another setting for performing theroutine when any one of one or more conditions is satisfied, wherein theone or more conditions include at least one condition selected from auser input condition, and the user input condition is satisfied when theuser has caused the operation to be dismissed during a particular timeselected from a time prior to the event and another time after theevent; accessing, subsequent to the first user input and the second userinput, data that provides an indication of whether the any one of theone or more conditions has been satisfied; determining, based onaccessing the data, that the data indicates that the any one of the oneor more conditions has been satisfied; and when the data indicates theuser input condition has been satisfied at the time prior to the event:causing performance of the operation to be dismissed, such that anoutput, to be provided by the computing device and corresponding to theoperation, is omitted when the event subsequently occurs, and causingthe automated assistant to initialize performance of the routine, priorto the event, in response to determining that the any one of the one ormore conditions has been satisfied.
 11. The method of claim 10, whereinthe data indicates that the user input condition has been satisfied by athird user input for causing the operation to be dismissed at the timeprior to the event.
 12. The method of claim 11, wherein the third userinput is a physical gesture, the operation is the computing deviceproviding an audible alarm output, and the event is a temporal arrivalof a particular clock time.
 13. The method of claim 10, furthercomprising: when the data indicates that the user input condition hasbeen satisfied at the other time after the event: causing the output,being provided by the computing device according to the setting, to bedismissed in response to a third user input from the user, and causing,in response to the third user input from the user, the automatedassistant to initialize performance of the routine at the other timeafter the event.
 14. The method of claim 10, wherein the one or moreconditions include at least one condition further selected from asituational condition, and the method further comprises: when the dataindicates that the situational condition has been satisfied at the timeprior to the event: causing performance of the operation to bedismissed, such that the output, to be provided by the computing deviceand corresponding to the operation, is omitted when the eventsubsequently occurs, and causing the automated assistant to initializeperformance of the routine, prior to the event, in response todetermining that the situational condition has been satisfied.
 15. Themethod of claim 14, wherein the situational condition includes at leastan input parameter that identifies a particular user input that, whenreceived at a separate computing device, causes the data to be generatedfor indicating that the situational condition has been satisfied. 16.The method of claim 15, wherein the particular user input corresponds toa spoken utterance, provided by the user, to an automated assistantinterface of the separate computing device, and the spoken utterancecharacterizes a request for the automated assistant to perform aparticular action that is omitted from the routine.
 17. A methodimplemented by one or more processors, the method comprising: receivingdata that characterizes a first operational status of a first computingdevice and a second operational status of a second computing device,wherein each of the first computing device and the second computingdevice provide access to an automated assistant that is configured to beresponsive to natural language inputs from a user and perform a routinethat includes initializing multiple different actions to be performed;determining, based on the data, that the first operational statusindicates the first computing device is scheduled to perform anoperation in response to at least an event condition being satisfied,and the second operational status indicates that the second computingdevice is to perform an action of an automated assistant routine inresponse to a condition being satisfied, wherein the condition is aparticular condition selected from a user input condition, a situationalcondition, and the event condition; receiving, subsequent to receivingthe data, other data that characterizes a presence of the user within anenvironment that includes the first computing device and the secondcomputing device; determining, based on the other data, whether thecondition has been satisfied; and when the situational condition isdetermined, based on the other data, to have been satisfied prior to anevent that corresponds to the event condition: causing performance ofthe operation to be dismissed, such that an output, to be provided bythe first computing device and corresponding to the operation, isomitted when the event subsequently occurs, and causing the automatedassistant to initialize performance of the routine, prior to the event,in response to determining that the situational condition has beensatisfied.
 18. The method of claim 17, further comprising: when theevent condition and the situational condition are determined, based onthe data, to be satisfied and the first computing device is providingthe output subsequent to the event: causing the first computing deviceto discontinue providing the output, and causing the automated assistantto initialize performance of the routine, prior to the event, inresponse to determining that the situational condition has beensatisfied.
 19. The method of claim 18, wherein the event is a temporalarrival of a clock time and the situational condition is satisfied when,at least, the presence of the user, characterized by the other data,indicates the user has changed locations within the environment.
 20. Themethod of claim 19, wherein the output is an audible output that isdirectly perceivable by the user, and is provided by the first computingdevice via a speaker that is configured to provide a natural languageoutput to the user during performance of the routine by the automatedassistant. 21-29. (canceled)